System and method for calculating location using a combination of odometry and landmarks

ABSTRACT

Disclosed is a system and method for calculation a location in a real-time manner using a combination of odometry and artificial landmarks. The system for calculating a location comprising a landmark detection unit detecting an image coordinates value of the artificial landmark corresponding to a location in a two-dimensional image coordinate system with respect to a mobile robot from an image obtained by photographing a specific space where the artificial landmarks are provided; a landmark identification unit comparing a predicted image value of the artificial landmark, obtained by converting a location coordinates value of the artificial landmark into an image coordinates value corresponding to the location in the two-dimensional image coordinate system with respect to a location coordinates value corresponding to a location in an actual three-dimensional spatial coordinate system of the mobile robot, with an image coordinates value detected by the landmark detection unit to detect the location coordinates value of the artificial landmark; a first location calculation unit calculating a current location coordinates value of the mobile robot using a predetermined location calculation algorithm based on the image coordinates value detected by the landmark detection unit and the location coordinates value detected by the landmark identification unit; a second location calculation unit calculating a current location coordinates value of the mobile robot using a predetermined location calculation algorithm based on odometry information of the mobile robot; and a main control unit updating the current location coordinates value of the mobile robot, using the location coordinates value calculated by the first location calculation unit when the location coordinates value calculated by the first location calculation unit exists, or using the location coordinate value obtained from the second location calculation unit when the location coordinates value calculated by the first location calculation unit does not exist.

CROSS-REFERENCE TO RELATED PATENT APPLICATION

This application claims the benefit of Korean Patent Application No.10-2006-70838, filed on Jul. 27, 2006, in the Korean IntellectualProperty Office, the disclosure of which is incorporated herein in itsentirety by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a system and method for calculating alocation using a combination of odometry and landmarks, and moreparticularly, to a system and method for calculating a location using acombination of odometry and landmarks in a real-time manner duringmovement of a mobile robot.

2. Description of the Related Art

In order to allow a mobile robot to make a routing plan to a destinationin an indoor room and perform automatic driving, the mobile robot mustrecognize its location in an indoor room beforehand. A method of usingan odometer of a wheel installed in the robot in order to obtainposition information is known in the art. In this method, a relativedistance and a relative direction are calculated with respect to aparticular position using a revolution velocity, a diameter, and abaseline of the wheel in order to obtain the location information. Thismethod of using odometry to obtain the location information has twosignificant problems. Firstly, since odometry is basically a relativelocation calculation method, an initial start location should be setbeforehand. Secondly, since odometry uses the revolution velocity of thewheel to measure distance, errors may occur when the wheel skids on aslippery ground surface depending on a ground condition. Althoughodometry provides relatively accurate location information over a shortdistance, errors are accumulated as a driving distance increases, and asolution for overcoming this problem has not been sufficiently studied.Thus, the location information cannot be reliably obtained using onlyodometry if there is no error correction method.

In some cases, artificial landmarks have been used as another means forrecognizing the location of the robot. In this method, artificiallandmarks discriminated from the background are distributedly providedin an indoor room, and image signals obtained by photographing theartificial landmarks using a camera installed in the robot are processedto recognize the artificial landmarks so that the current location ofthe robot can be obtained. The location of the robot is calculated byreferring to the image coordinates of the recognized landmark andcoordinate information that has been previously stored for thecorresponding landmark.

In order to calculate the location based on landmarks, a predeterminednumber of landmarks should come within the field of view of the camera.Generally, since the viewing angle of the camera is limited, an area inwhich the location can be obtained using the landmarks is limited. Forthis reason, in order to enable location information to be obtained atany location in the entire indoor room based on landmarks, the landmarksshould be distributed close enough together so that a required number oflandmarks are in the field of view of the camera at an arbitraryposition in an indoor room. It is not easy to arrange the landmarks tosatisfy this condition, and in particular, this method is considered tobe very inefficient from the viewpoint of cost, time, and aestheticalappearance when a location measurement system is constructed in a widearea space such as a market or a public building using only artificiallandmarks. In addition, when the landmark is temporarily obscured byobstacles such as visitors or customers, the location information cannotbe appropriately obtained.

Typical artificial landmarks include geometric patterns such as circularand rectangular shapes or barcodes that can be discriminated from thebackground. In order to calculate the location of the robot, a processof recognizing these kinds of patterns should be performed beforehand.Also, since the image signal input through the camera is influenced byvarious conditions such as a distance between the landmark and thecamera, a direction, and illumination, it is difficult to obtain stablerecognition performance in a common indoor environment. In particular,since image signals become weak during the night, it is nearlyimpossible to perform the process of recognizing patterns based on imageprocessing during the night.

In order to overcome the aforementioned problem in image processing, amethod of using a predetermined wavelength band of light beams has beenproposed. In this method, a light source capable of irradiating apredetermined wavelength band of light beams, such as an infrared lightemitting diode (IR-LED), is used as the artificial landmark, and anoptical filter capable of transmitting only the corresponding wavelengthband is installed in the camera, so that only signals irradiated fromthe light sources of landmarks can be captured in the camera image.Accordingly, the image processing procedure for detecting artificiallandmarks can be simplified, and recognition reliability can also beimproved. However, since these light sources of the landmarks do nothave different shapes, the landmarks should be appropriatelydiscriminated from one another. In order to discriminate the lightsources of the landmarks, a method of detecting the landmarks bysequentially turning on/off the light sources has been proposed.However, the process of sequentially turning on/off the light sourcesrequires a great amount of time which increases in proportion to thenumber of landmarks. Also, the location information cannot be providedin a real-time manner as the recognition of the landmarks should beperformed in a state in which the robot pauses. Therefore, this methodcannot be applied during the movement.

As described above, both the conventional methods using odometry and theartificial landmarks have some shortcomings. Although the method ofusing odometry provides high accuracy over a short distance, errors maybe accumulated as the driving distance increases due to the relativelocation measurement. Therefore, this method is not considered to beeasy. On the other hand, although the method of using the artificiallandmarks provides absolute location information supposing the landmarksare successfully detected, the location information may not be obtainedwhen the detection of landmarks fails due to obstacles. Also, if thespace to be covered by the location measurement system increases,arranging new landmarks is burdensome.

In addition, the method of using the artificial landmarks is sensitiveto external conditions such as illumination and cannot reliably detectlandmarks when the landmarks are detected through a pattern recognitionprocess using discriminatable patterns as the artificial landmarks. Evenwhen light sources and an optical filter are adopted in the landmarksand the camera, respectively, in order to solve this problem, it isdifficult to discriminate the landmarks from one another even if thelandmarks are appropriately detected.

SUMMARY OF THE INVENTION

The present invention provides a system and method for calculatinglocation information using a combination of an artificial landmark basedlocation calculation method and an odometry based location calculationmethod, by which successive location information can be calculated usingonly a little number of landmarks over any wide area indoor roomregardless of a landmark detection failure or temporary landmarkobscurity.

In addition, the present invention provides a technology of identifyingthe landmark all day long in a real-time manner using light sources ofthe landmarks and an optical filter regardless of changes in externalconditions such as illumination without controlling an on/off operationof the light source of the landmark.

According to an aspect of the present invention, there is provided asystem for calculating a location in a real-time manner using acombination of odometry and artificial landmarks, the system comprising:a landmark detection unit detecting an image coordinates value of theartificial landmark corresponding to a location in a two-dimensionalimage coordinate system with respect to a mobile robot from an imageobtained by photographing a specific space where the artificiallandmarks are provided; a landmark identification unit comparing apredicted image value of the artificial landmark, obtained by convertinga location coordinates value of the artificial landmark into an imagecoordinates value corresponding to the location in the two-dimensionalimage coordinate system with respect to a location coordinates valuecorresponding to a location in an actual three-dimensional spatialcoordinate system of the mobile robot, with an image coordinates valuedetected by the landmark detection unit to detect the locationcoordinates value of the artificial landmark; a first locationcalculation unit calculating a current location coordinates value of themobile robot using a predetermined location calculation algorithm basedon the image coordinates value detected by the landmark detection unitand the location coordinates value detected by the landmarkidentification unit; a second location calculation unit calculating acurrent location coordinates value of the mobile robot using apredetermined location calculation algorithm based on odometryinformation of the mobile robot; and a main control unit updating thecurrent location coordinates value of the mobile robot, using thelocation coordinates value calculated by the first location calculationunit when the location coordinates value calculated by the firstlocation calculation unit exists, or using the location coordinate valueobtained from the second location calculation unit when the locationcoordinates value calculated by the first location calculation unit doesnot exist.

The artificial landmark may include a light source, such as anelectroluminescent device or a light emitting diode, which has uniqueidentification information and can irradiate a particular wavelengthband of light beams.

The landmark detection unit may include a camera having an opticalfilter capable of transmitting only a specific wavelength band of lightbeams irradiated by the light source included in the artificiallandmark.

The mobile robot may include a landmark control unit generating anON/OFF signal for selectively turning on/off the light sources of theartificial landmarks, and each of the landmarks may include a lightsource control unit receiving a signal from the landmark control unitand controlling turning on/off the light sources.

At least two artificial landmarks are provided within an area in whichthe mobile robot moves.

The main control unit may control a camera included in the mobile robotor an odometer sensor by transmitting signals through wired or wirelesscommunication.

The main control unit may repeat processes of updating a currentlocation coordinates value of the mobile robot, receiving the locationcoordinates value obtained from the first or second location calculationunit, and updating the current location coordinates value of the mobilerobot again.

The landmark detection unit calculates the image coordinate value of theartificial landmark on the basis of the image coordinates valuesobtained by regarding the mobile robot as a center point of thetwo-dimensional image coordinate system

The landmark identification unit may calculate a deviation between thepredicted image value of the artificial landmark and the imagecoordinates value detected by the landmark detection unit, and maycalculate a location coordinate value of the artificial landmarkcorresponding to an image coordinates value having the least deviation.

The first location calculation unit may calculate a scaling factor, afactor for a two-dimensional circulation, and a two-dimensionalhorizontal shifting constant that are required to convert the imagecoordinate system into the spatial coordinate system using the imagecoordinates value detected by the landmark detection unit and thelocation coordinates value detected by the landmark identification unit,and may convert the image coordinates value of the mobile robotcorresponding to a center point of the two-dimensional image coordinatesystem into a location coordinates value of the spatial coordinatesystem.

The second location calculation unit may measure a movement velocity ofthe mobile robot using a wheel sensor attached to a wheel of the mobilerobot, and calculate a current location coordinates value of the mobilerobot on the basis of a moving distance corresponding to the movementvelocity.

According to another aspect of the present invention, there is provideda method of calculating a location in a real-time manner using acombination of odometry and artificial landmarks, the method comprising:(a) detecting a image coordinates value of the artificial landmarkcorresponding to a location in a two-dimensional image coordinate systemwith respect to a mobile robot from an image obtained by photographing aspecific space where the artificial landmarks are provided; (b)comparing a predicted image value of the artificial landmark, obtainedby converting a location coordinates value of the artificial landmarkinto an image coordinates value corresponding to the location in thetwo-dimensional image coordinate system with respect to a locationcoordinates value corresponding to a location in an actualthree-dimensional spatial coordinate system of the mobile robot, with animage coordinates value detected by the landmark detection unit todetect the location coordinates value of the artificial landmark; (c)calculating a current location coordinates value of the mobile robotusing a predetermined location calculation algorithm based on the imagecoordinates value detected in the (a) detection of the image coordinatesvalue and the location coordinates value detected in the (b) comparisonof the predicted image value; (d) calculating a current locationcoordinates value of the mobile robot using a predetermined locationcalculation algorithm based on odometry information of the mobile robot;and (e) updating the current location coordinates value of the mobilerobot using the location coordinates value calculated in the (c)calculation of the current location coordinates value when the locationcoordinates value calculated in the (c) calculation of the currentlocation coordinates value exists, or using the location coordinatevalue obtained in the (d) calculation of the current locationcoordinates value when the location coordinates value calculated in the(c) does not exist.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other features and advantages of the present inventionwill become more apparent by describing in detail exemplary embodimentsthereof with reference to the attached drawings in which:

FIG. 1 is a block diagram illustrating components of a system forcalculating a location using a combination of odometry and landmarks ina real-time manner according to an exemplary embodiment of the presentinvention;

FIG. 2 is a schematic diagram for describing a process of photographinglandmarks performed by a mobile robot according to an exemplaryembodiment of the present invention;

FIG. 3A is a photograph taken by a typical camera installed in a mobilerobot;

FIG. 3B is a photograph taken by a camera installed in a mobile robotusing an optical filter according to an exemplary embodiment of thepresent invention;

FIG. 4 is a flowchart illustrating a process of calculating a locationusing a combination of odometry and artificial landmarks in a real-timemanner according to an exemplary embodiment of the present invention;and

FIG. 5 is a graph for describing a relationship for transformationbetween a spatial coordinate system and an image coordinate systemaccording to an exemplary embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Hereinafter, exemplary embodiments of the present invention will bedescribed in detail with reference to the accompanying drawings.

FIG. 1 is a block diagram illustrating components of a system forcalculating a location of a mobile robot in a real-time manner accordingto an exemplary embodiment of the present invention. FIG. 2 is aschematic diagram for describing a process of photographing landmarksperformed by a mobile robot according to an exemplary embodiment of thepresent invention. FIG. 3A is a photograph taken by a typical camerainstalled in a mobile robot, and FIG. 3B is a photograph taken by acamera installed in a mobile robot using an optical filter according toan exemplary embodiment of the present invention;

FIG. 2 shows a process of obtaining images in a landmark detection unit100 of FIG. 1, and FIGS. 3A and 3B show images obtained through theprocess of FIG. 2. They will be described in association with FIG. 1.

Referring to FIG. 1, the system according to an exemplary embodiment ofthe present invention includes a landmark detection unit 100, a landmarkidentification unit 110, a first location calculation unit 120, a secondlocation calculation unit 130, a main control unit 140, and a landmarkcontrol unit 150.

The landmark control unit 150 controls on/off operations of lightsources of landmarks. Each light source has a light emitting diode (LED)or an electroluminescent element, capable of emitting light beams havinga specific wavelength and brightness of, which can be turned on and offaccording to a signal through an external communication control module.In addition, each landmark has a unique identification (ID) todiscriminate it from other landmarks, and is usually attached to aceiling in a work space. The location at which the landmark is attachedin the work space (i.e., spatial coordinates) is stored through anactual measurement.

The landmark detection unit 100 detects image coordinates of thelandmarks from an image signal input through the camera. Preferably, thecamera has an optical filter for transmitting only a predeterminedwavelength of light beams irradiated from the light source of thecorresponding landmark. In addition, as shown in FIG. 2, the camera isinstalled in a mobile robot in such a way that a lens of the cameraviews the ceiling, and an optical axis of the camera is perpendicular tothe ground surface. Since the optical filter installed in the camera isdesigned to transmit only light beams having the same wavelength as thatof the light source of the landmark, an image shown in FIG. 3B isobtained. As a result, it is possible to simplify an image processingprocedure for detecting landmarks and allow the landmarks to be detectedall day long regardless of changes in external conditions such asillumination.

The landmark identification unit 110 identifies the detected landmarks.Since the optical filter for transmitting only light beams having thesame wavelength band as that of the light sources of the landmarks isused in the image processing procedure for detecting the landmarks, theimage processing procedure for detecting the landmarks is simplyperformed by detecting regions having a brightness equal to or higherthan a predetermined critical value through a binarization processperformed on the image signals. The image coordinates of the landmarkobtained by detecting the light source are determined as the coordinatesof the center point of the detected region through binarization.

The first location calculation unit 120 calculates spatial coordinatesand a direction of the robot on the basis of the detected imagecoordinates of the landmark and the spatial coordinates previouslystored for the corresponding landmark in a work space.

When it is determined that the number of detected landmarks issufficient to calculate the location of the robot through the landmarkdetection process, the location of the robot is calculated on the basisof the image coordinates of the detected landmarks and the previouslystored spatial coordinates of the corresponding landmark. In order torefer to the spatial coordinates of the landmarks, the detectedlandmarks should be identified beforehand. The following descriptionrelates to only a location calculation process performed when thedetected landmark is identified. A more detailed description of thelocation calculation process will be given below in association withFIG. 3.

In order to calculate the location of the robot, at least two landmarksshould be detected. At least three landmarks may be used to minimizeerrors in the location calculation. Assume that, for two detectedlandmarks L_(i), and L_(j), the detected image coordinates are (x_(i),y_(i)) and (x_(j), y_(j)), and the previously stored spatial coordinatesare (X_(i), Y_(i), Z_(i)), (X_(j), Y_(j), Z_(j)), respectively. In thiscase, the Z-axis coordinates denote the vertical distance between thecamera and the landmarks, and are necessary to obtain accurate locationinformation despite variations in the height of the ceiling over anindoor room where the robot is driven. In other words, the methodaccording to the present invention may not require separate ceilingheight information (i.e., Z-axis coordinates) if the height of theceiling is constant.

First of all, image coordinates (p_(i), q_(i)) and (p_(j), q_(j))obtained by correcting distortion of a camera lens from the originalimage coordinates are calculated as follows:

$\begin{bmatrix}x_{d} \\y_{d} \\1\end{bmatrix} = \left. {\begin{bmatrix}f_{x} & 0 & c_{x} \\0 & f_{y} & c_{y} \\0 & 0 & 1\end{bmatrix}^{- 1}\begin{bmatrix}x \\y \\1\end{bmatrix}}\Rightarrow\left\{ {{{\begin{matrix}{x_{d} = {\left\lbrack {x - c_{x}} \right\rbrack/f_{x}}} \\{y_{d} = {\left\lbrack {y - c_{y}} \right\rbrack/f_{y}}}\end{matrix}\begin{bmatrix}x_{u} \\y_{u}\end{bmatrix}} = {\frac{1}{1 + {k_{1}r^{2}} + {k_{2}r^{4}} + {k_{3}r^{6}}}\begin{bmatrix}x_{d} \\y_{d}\end{bmatrix}}},{r^{2} = {{x_{u}^{2} + {y_{u}^{2}\begin{bmatrix}p \\q \\1\end{bmatrix}}} = {\begin{bmatrix}f_{x} & 0 & c_{x} \\0 & f_{y} & c_{y} \\0 & 0 & 1\end{bmatrix}\begin{bmatrix}x_{u} \\y_{u} \\1\end{bmatrix}}}}} \right. \right.$

where, f_(x) and f_(y) denote focal distances; c_(x) and c_(y) denoteinternal camera parameters indicating image coordinates of the centerpoint of the lens; k₁, k₂, and k₃ denote lens distortion coefficientscorresponding to variables obtained through a camera calibrationprocess. In this equation, the detected image coordinates value isdesignated as (x, y), and the coordinates value obtained by correctingthe distortion of the lens is designated as (p, q) without indexing anysubscript for discriminating the two landmarks.

The camera lens distortion correction is necessary to calculate accuratelocations. Particularly, this process is indispensable in a case where afisheye lens is used to enlarge the field of view of the camera. Sincelens distortion correction is performed only for the image coordinatesof the detected landmarks, additional processing time is not needed.

Subsequently, a height normalization process is performed in order toremove image coordinates variations caused by the height difference ofthe ceiling to which the landmarks are attached. As described above,this process may be omitted when the height of the ceiling is constantover the entire indoor room. The image coordinates of the landmarkattached to the ceiling are in inverse proportion to the height of theceiling, so that the image coordinates of the landmark in a farther thepoint of origin as the height of the ceiling increases, while the imagecoordinates of the landmark in a nearer the point of origin as theheight of the ceiling is reduced. Therefore, the image coordinates(u_(i), v_(i)) and (u_(j), v_(j)) normalized to a reference height hfrom the distortion-corrected image coordinates (p_(i), q_(i)) and(p_(j), q_(j)) can be obtained as follows:

$\left( {u_{i},v_{i}} \right) = {\frac{Z_{i}}{h}\left( {p_{i},q_{i}} \right)}$${\left( {u_{j},v_{j}} \right) = {\frac{Z_{j}}{h}\left( {p_{j},q_{j}} \right)}},$

where, h denotes an arbitrary positive constant.

Subsequently, location information (r_(x), r_(y), θ) of the robot iscalculated using the image coordinates (u_(i), v_(i)) and (u_(j), v_(j))obtained by performing the distortion correction and the heightnormalization for the detected landmarks and the stored spatialcoordinates (X_(i), Y_(i), Z_(i)) and (X_(j), Y_(j), Z_(j)), where θdenotes a heading angle of the robot with respect to the Y-axis of thespatial coordinate system.

Since the camera views the ceiling at a perpendicular angle, it isassumed that the robot is located in the center of the image. In otherwords, the image coordinates of the robot become (c_(x), c_(y)). Thespatial coordinates (r_(x), r_(y)) of the robot can be obtained bytransforming the image coordinates of the landmarks L₁, L₂ into spatialcoordinates and applying them to the image coordinates (c_(x), c_(y)).Since the camera views the ceiling at a perpendicular angle, thecoordinate system transformation may be performed through scaletransformation, two-dimensional circulation, and two-dimensionalhorizontal shifting as follows:

$s = {\frac{D}{d} = \frac{\sqrt{\left( {X_{i} - X_{j}} \right)^{2} + \left( {Y_{i} - Y_{j}} \right)^{2}}}{\sqrt{\left( {u_{i} - u_{j}} \right)^{2} + \left( {v_{i} - v_{j}} \right)^{2}}}}$${\cos \; \theta} = \frac{{\left( {u_{j} - u_{i}} \right)\left( {X_{j} - X_{i}} \right)} + {\left( {v_{j} - v_{i}} \right)\left( {Y_{j} - Y_{i}} \right)}}{dD}$${{\sin \; \theta} = {{\frac{{\left( {u_{j} - u_{i}} \right)\left( {Y_{j} - Y_{i}} \right)} - {\left( {v_{j} - v_{i}} \right)\left( {X_{j} - X_{i}} \right)}}{dD}\begin{bmatrix}r_{x} \\r_{y}\end{bmatrix}} = {{{s\begin{bmatrix}{\cos \; \theta} & {{- \sin}\; \theta} \\{\sin \; \theta} & {\cos \; \theta}\end{bmatrix}}\left( {\begin{bmatrix}c_{x} \\c_{y}\end{bmatrix} - \begin{bmatrix}u_{i} \\v_{i}\end{bmatrix}} \right)} + \begin{bmatrix}X_{i} \\Y_{i}\end{bmatrix}}}},$

where, the scaling factor s is constant regardless of a pair of thelandmarks used in the location calculation, and the value obtained byperforming initial location calculation is stored in a memory unit inorder to predict image coordinates for a subsequent landmarkidentification process.

In the above method, the location of the robot is calculated on thebasis of the image coordinates obtained from the image and thepreviously stored spatial coordinates of the landmarks. If this methodis reversely applied, the image coordinates can be obtained from thecamera image on the basis of the location of the robot and the spatialcoordinate of the corresponding landmark.

Assume that the current location of the robot is (r_(x), r_(y), θ), andthe spatial coordinates of the landmark L_(k) are (X_(k), Y_(k), Z_(k)),where k=1, and n denotes the total number of attached landmarks. Then,predicted image coordinates (û_(k), {circumflex over (v)}_(k)) of thelandmark L_(k) without considering the lens distortion and the heightnormalization can be calculated as follows:

${\begin{bmatrix}{\hat{u}}_{k} \\{\hat{v}}_{k}\end{bmatrix} = {\begin{bmatrix}c_{x} \\c_{y}\end{bmatrix} - {{\frac{1}{s}\begin{bmatrix}{\cos \; \theta} & {\sin \; \theta} \\{{- \sin}\; \theta} & {\cos \; \theta}\end{bmatrix}}\left( {\begin{bmatrix}r_{x} \\r_{y}\end{bmatrix} - \begin{bmatrix}X_{k} \\Y_{k}\end{bmatrix}} \right)}}},{k = 1},\ldots \mspace{14mu},{n.}$

If the height of the landmark and the lens distortion are reflected onthe calculated image coordinates (û_(k), {circumflex over (v)}_(k)),finally predicted image coordinates ({circumflex over (x)}_(k), ŷ_(k))can be obtained as follows:

${\begin{bmatrix}{\hat{p}}_{k} \\{\hat{q}}_{k} \\1\end{bmatrix} = {{{\frac{h}{Z_{k}}\begin{bmatrix}{\hat{u}}_{k} \\{\hat{v}}_{k} \\1\end{bmatrix}}\begin{bmatrix}{\hat{x}}_{u\_ k} \\{\hat{y}}_{u\_ k} \\1\end{bmatrix}} = {{{\begin{bmatrix}f_{x} & 0 & c_{x} \\0 & f_{y} & c_{y} \\0 & 0 & 1\end{bmatrix}^{- 1}\begin{bmatrix}{\hat{p}}_{k} \\{\hat{q}}_{k} \\1\end{bmatrix}}\begin{bmatrix}{\hat{x}}_{d\_ k} \\{\hat{y}}_{d\_ k}\end{bmatrix}} = {\left( {1 + {k_{1}r^{2}} + {k_{2}r^{4}} + {k_{3}r^{6}}} \right)\begin{bmatrix}{\hat{x}}_{u\_ k} \\{\hat{y}}_{u\_ k}\end{bmatrix}}}}},{r^{2} = {{{\hat{x}}_{u\_ k}^{2} + {{\hat{y}}_{u\_ k}^{2}\begin{bmatrix}{\hat{x}}_{k} \\{\hat{y}}_{k} \\1\end{bmatrix}}} = {{\begin{bmatrix}f_{x} & 0 & c_{x} \\0 & f_{y} & c_{y} \\0 & 0 & 1\end{bmatrix}\begin{bmatrix}{\hat{x}}_{d\_ k} \\{\hat{y}}_{d\_ k} \\1\end{bmatrix}}.}}}$

The second location calculation unit 130 calculates location informationon the basis of the odometry. Preferably, the mobile robot has a sensorfor obtaining odometry information, such as an encoder.

The location information using odometry is calculated on the basis ofthe movement velocities of both wheels of the robot. The movementvelocities of the wheels are measured using wheel sensors attached toeach wheel. Assuming that the location of the robot at a time point(t−1) is (r_(x) ^(t−1), r_(y) ^(t−1), θ^(t−1)) the movement velocitiesof both wheels at a time point t are v_(l) and v_(r), the wheel baselineis w, and the interval between the time points t and t−1 is Δt, thelocation of the robot (r_(x) ^(t), r_(y) ^(t), θ^(t)) at the time pointt can be calculated as follows:

$r_{x}^{t} = {r_{x}^{t - 1} + {\Delta \; T\; {\cos \left( {\theta^{t - 1} + \frac{\Delta \; R}{2}} \right)}}}$$r_{y}^{t} = {r_{y}^{t - 1} + {\Delta \; T\; {\sin \left( {\theta^{t - 1} + \frac{\Delta \; R}{2}} \right)}}}$${\theta^{t} = {\theta^{t - 1} + {\Delta \; R}}},{where},{{\Delta \; T} = {\frac{v_{l} + v_{r}}{2}\Delta \; t}}$${\Delta \; R} = {\frac{v_{r} - v_{l}}{W}\Delta \; {t.}}$

The location information calculated at the time point t is used tocalculate the location information at the time point t+1. Locationinformation at the time point when the landmark based locationcalculation fails is used as initial location information.

The main control unit 140 stores the spatial coordinates of the landmarkin a work space and a camera lens distortion coefficient and entirelycontrols each module, so that successive location information iscalculated while switching between a landmark mode and an odometry modeis automatically performed.

FIG. 4 is a flowchart for describing a process of calculating a locationof a mobile robot in a real-time manner according to an exemplaryembodiment of the present invention.

FIG. 5 is a graph for describing a relationship of transformationbetween a spatial coordinate system and an image coordinate systemaccording to an exemplary embodiment of the present invention.

FIG. 5 simultaneously shows the spatial coordinate system and the imagecoordinate system used for transformation between the image coordinatesand the spatial coordinates in the process of calculating the locationof the robot in a real-time manner, which will be described in moredetail in association with FIG. 4.

First of all, initial location information of the mobile robot iscalculated while the on/off operations of artificial landmarks arecontrolled. The initial location calculation is performed at the placewhere the landmark is provided. After the initial location iscalculated, the location information is updated in a real-time mannerthrough a landmark based location calculation process as long as anylandmark is continuously detected within the field of view of the camera(i.e., in an artificial landmark mode). When the detection of thelandmark fails, for example, when a landmark disappears out of the fieldof view of the camera as the robot moves, or when the landmark istemporarily obscured by obstacles, the location calculation process isswitched to an odometry mode, and then, subsequent location informationis calculated using odometry (in an odometry mode). In the odometrymode, it is determined whether or not a landmark is detected in thecamera image in every location update period. When a landmark is notdetected, the location information is continuously calculated in anodometry mode. Otherwise, when a landmark is detected, the locationinformation is calculated in an artificial landmark mode.

The initial location recognition is done to recognize the location ofthe robot in an indoor room when the robot does not initially have anylocation information at all. Since there is no information on thelocation of the robot when the initial location is calculated, it isimpossible to identify the detected landmark through only imageprocessing. Therefore, the landmark is identified in the initiallocation calculation process using a conventional control method (i.e.,by sequentially turning on/off the light sources of the landmarks).Specifically, only one of a plurality of light sources provided in anindoor room is turned on while other light sources are turned off. Thelight source turn-on command may be issued by transmitting a turn-onsignal to the corresponding light source through a landmark controlmodule. Then, the image is obtained using the camera, and the lightsource is detected in the image, so that the detected light source isidentified as the landmark transmitting the turn-on signal through thelandmark control module. Subsequently, the next landmark is selected,and the turn-on signal is transmitted, so that the landmark detectionprocess is repeated until the number of detected landmarks is sufficientto calculate the location of the robot.

When the number of detected landmarks is sufficient to calculate thelocation, and they are identified through the above process, the initiallocation information of the robot is calculated by applying theaforementioned landmark based location calculation method. It should benoted that the initial location recognition should be performed in theplace where the landmarks are provided.

Although this initial location recognition process takes time as theprocess of obtaining the image and detecting the landmarks should beperformed while the light sources of the landmarks are sequentiallyturned on and off in a state in which the robot pauses its movement, theoverall driving of the robot is not influenced because this process isperformed only once when the robot is initialized.

Alternatively, the robot may be controlled to always start to drive at aspecific location, and this specific location may be set as an initiallocation. For example, considering that an electrical charger system isnecessary to operate the robot, the location of the electrical chargersystem may be set as the initial location.

In a landmark based location update operation, an image is obtained fromthe camera at a predetermined time interval, the landmarks are detectedin the obtained image, and the location information of the robot isupdated (in an artificial landmark mode). The photographing speed may bedetermined depending on the camera. For example, when a typical camerathat can obtain 30 image frames per second is used, the location updateperiod of the robot can be set to 30 Hz. This process is continuouslyperformed as long as the landmark is detected within the field of viewof the camera.

Now, how to update the location information of the robot on the basis ofthe artificial landmarks during the driving will be described. First ofall, it is assumed that the number of landmarks detected in the cameraimage is sufficient to calculate the location of the robot within thecurrent location update period (at a time point t). In order to identifythe detected landmarks, the image coordinates of every landmark providedin an indoor room are predicted on the basis of the location of therobot in the most previous time point (e.g., t−1). The aforementionedmethod of predicting the image coordinates may be used in this case.Since the predicted image coordinates are not calculated on the basis ofthe current location but the most previous location of the robot, thepredicted image coordinates may be deviated from the current imagecoordinates if the robot moves for the time interval between the timepoints t−1 and t. However, since the driving speed of the robot islimited, and the photographing speed of the camera is sufficiently fast,the deviation is not very large.

For example, assuming that the photographing speed of the camera is setto 30 frames per second, and a movement velocity of the robot is set to3 m/s, the robot physically moves 10 cm for the shot-to-shot interval ofthe camera. Therefore, only several pixels in an image are changed,considering the height of a typical ceiling. As a result, the detectedlandmark can be identified in such a way that the image coordinates ofthe landmark detected to be closest to the most previous location of thecorresponding landmark in the current image can be predicted as theimage coordinates of the current location of the landmark.

Alternatively, the landmarks detected in the camera image in the currentlocation update period can be identified as follows.

The neariest landmark with the robot out of landmark detected in theupdate period just before preseut location becomes an equal landmark andit pursues,

When the most previously detected landmark is disappeared from the fieldof view of the camera, and a new landmark is detected within the fieldof view of the camera as the robot moves, the new landmark can beidentified using the aforementioned image coordinates prediction method.

When the detected landmark is identified as described above, thelocation information of the robot is calculated using the aforementionedlandmark based location calculation method by referring to the spatialcoordinates previously stored for the corresponding landmark, and thenthe current location information is updated using this coordinateinformation. The updated location information is used to predict theimage coordinates in the subsequent process of the location informationupdate period. This process is repeated as long as landmarks aredetected, so as to provide the location information in a real-timemanner.

If the landmark detection fails when the landmark disappears from thefield of view of the camera or is obscured by any obstacle as the robotmoves, the calculation mode is changed to the odometry mode, and thesubsequent location information is calculated using odometry.

In the odometry mode, the camera image is obtained in every locationinformation update period to inspect whether or not the landmark isdetected while the location information is updated using theaforementioned odometry information. When the landmark is detectedwithin the field of view of the camera, the calculation modeautomatically returns to the artificial landmark mode using thefollowing method.

When the landmark is detected within the camera image while the locationinformation of the robot is calculated in the odometry mode, the imagecoordinates of each landmark are predicted on the basis of the robotlocation information calculated using odometry. The prediction of theimage coordinates is performed using the aforementioned landmark imagecoordinates prediction method. The detected landmark is identified in asimilar way to the landmark based location update method, in which theimage coordinates closest to the image coordinates of the detectedlandmark are predicted as the current image coordinates of the landmark.If the landmark is identified, the location is calculated using thelandmark based location calculation method, and the current location isupdated. Subsequent location information is calculated in the artificiallandmark mode.

When the image coordinates are predicted on the basis of the locationinformation obtained in the odometry mode, a deviation between thepredicted image coordinates of the landmark and the actual imagecoordinates occurs due to errors in the odometer. However, sinceodometry provides relatively accurate location information over a shortdistance, it is possible to successfully identify the landmark as longas the driving distance is not too long.

The invention can also be embodied as computer readable codes on acomputer readable recording medium. The computer readable recordingmedium is any data storage device that can store data which can bethereafter read by a computer system. Examples of the computer readablerecording medium include read-only memory (ROM), random-access memory(RAM), CD-ROMs, magnetic tapes, floppy disks, optical data storagedevices, and carrier waves (such as data transmission through theInternet). The computer readable recording medium can also bedistributed over network coupled computer systems so that the computerreadable code is stored and executed in a distributed fashion.

According to the present invention, a location measurement systemcapable of covering a wide area can be constructed with only a littlenumber of landmarks using odometry. Therefore, it is possible to reducecost and time for constructing the location measurement system. Also, itis possible to provide a safe location measurement system capable ofproviding location information even when a landmark based locationcalculation fails.

In addition, according to the present invention, the robot is notrequired to stop to recognize its location, and it is possible tosuccessively provide real-time location information over any wide indoorarea by attaching only a little number of landmarks regardless ofobscurity or failure in detecting the landmarks. Therefore, it ispossible to allow the mobile robot to safely recognize its location atany place in an indoor room, accordingly make a routing plan to adesired destination, and freely drive.

Furthermore, according to the present invention, it is possible toconstruct a location measurement system capable of providing locationinformation regardless of a size or a structure of an indoor environmentsuch as the height of the ceiling. Therefore, it is possible to use themobile robot in a variety of indoor environments and widen a range ofrobot service applicability.

Still furthermore, according to the present invention, since thelocation information can be successively calculated in a real-timemanner regardless of a driving condition of the robot, it is possible todynamically change the routing plan on the basis of the calculatedlocation information. Therefore, it is possible to smoothly controlmotion of the robot, dynamically avoid obstacles, and change adestination during the driving.

Still furthermore, according to the present invention, the robot is notrequired to stop its movement or delay separate time to recognize itslocation. Therefore, it is possible to improve work efficiency in a workspace.

Still furthermore, according to the present invention, it is possible toprovide the location information all day long regardless of changes inthe external environment such as illumination. Therefore, it is possibleto safely drive the robot, and particularly, a robot patrol service canbe provided in the night.

Still furthermore, in the method of identifying the landmark using theimage coordinates prediction according to the present invention, it ispossible to verify whether or not the recognition is appropriatelyperformed by comparing the image coordinates of the landmark recognizedin the image processing with the predicted image coordinates even whengeometrical or natural landmarks are used instead of light sources.Therefore, it is possible to improve the reliability of a typicallandmark based location calculation system.

The location calculation system according to the present invention canbe applied to other devices or appliances that have been manuallycarried as well as a mobile robot.

According to the present invention, absolute coordinates in an indoorroom are provided in a real-time manner. Therefore, it is possible tomore accurately draw an environmental map by reflecting the datameasured on the basis of absolute location information providedaccording to the present invention when an environmental map for theindoor environment is created using ultrasonic, infrared, or visionsensors.

While the present invention has been particularly shown and describedwith reference to exemplary embodiments thereof, it will be understoodby those of ordinary skill in the art that various changes in form anddetails may be made therein without departing from the spirit and scopeof the present invention as defined by the following claims.

1. A system for calculating a location in a real-time manner using acombination of odometry and artificial landmarks, the system comprising:a landmark detection unit detecting an image coordinates value of anartificial landmark corresponding to a location in a two-dimensionalimage coordinate system with respect to a mobile robot from an imageobtained by photographing a specific space where the artificiallandmarks are provided; a landmark identification unit comparing apredicted image value of the artificial landmark, obtained by convertinga location coordinates value of the artificial landmark into an imagecoordinates value corresponding to the location in the two-dimensionalimage coordinate system with respect to a location coordinates valuecorresponding to a location in an actual three-dimensional spatialcoordinate system of the mobile robot, with an image coordinates valuedetected by the landmark detection unit to detect the locationcoordinates value of the artificial landmark; a first locationcalculation unit calculating a current location coordinates value of themobile robot using a predetermined location calculation algorithm basedon the image coordinates value detected by the landmark detection unitand the location coordinates value detected by the landmarkidentification unit; a second location calculation unit calculating acurrent location coordinates value of the mobile robot using apredetermined location calculation algorithm based on odometryinformation of the mobile robot; and a main control unit updating thecurrent location coordinates value of the mobile robot, using thelocation coordinates value calculated by the first location calculationunit when the location coordinates value calculated by the firstlocation calculation unit exists, or using the location coordinate valueobtained from the second location calculation unit when the locationcoordinates value calculated by the first location calculation unit doesnot exist.
 2. The system of claim 1, wherein the artificial landmarkincludes a light source, such as an electroluminescent device or a lightemitting diode, which has unique identification information and canirradiate a particular wavelength band of light beams.
 3. The system ofclaim 2, wherein the landmark detection unit includes a camera having anoptical filter capable of transmitting only a specific wavelength bandof light beams irradiated by the light source included in the artificiallandmark.
 4. The system of claim 2, wherein the mobile robot includes alandmark control unit generating an ON/OFF signal for selectivelyturning on/off the light sources of the artificial landmarks, and eachof the landmarks includes a light source control unit receiving a signalfrom the landmark control unit and controlling turning on/off the lightsources.
 5. The system of claim 1, wherein at least two artificiallandmarks are provided within an area in which the mobile robot moves.6. The system of claim 1, wherein the main control unit controls acamera included in the mobile robot or an odometer sensor bytransmitting signals through wired or wireless communication.
 7. Thesystem of claim 1, wherein the main control unit repeats processes ofupdating a current location coordinates value of the mobile robot,receiving the location coordinates value obtained from the first orsecond location calculation unit, and updating the current locationcoordinates value of the mobile robot again.
 8. The system of claim 1,wherein the landmark detection unit calculates the image coordinatevalue of the artificial landmark on the basis of the image coordinatevalues obtained by regarding the mobile robot as a center point of thetwo-dimensional image coordinate system.
 9. The system of claim 1,wherein the landmark identification unit calculates a deviation betweenthe predicted image value of the artificial landmark and the imagecoordinates value detected by the landmark detection unit, andcalculates a location coordinate value of the artificial landmarkcorresponding to an image coordinates value having the least deviation.10. The system of claim 1, wherein the first location calculation unitcalculates a scaling factor, a factor for a two-dimensional circulation,and a two-dimensional horizontal shifting constant that are required toconvert the image coordinate system into the spatial coordinate systemusing the image coordinates value detected by the landmark detectionunit and the location coordinates value detected by the landmarkidentification unit, and the first location calculation unit convertsthe image coordinate value of the mobile robot corresponding to a centerpoint of the two-dimensional image coordinate system into a locationcoordinates value of the spatial coordinate system.
 11. The system ofclaim 1, wherein the second location calculation unit measures amovement velocity of the mobile robot using a wheel sensor attached to awheel of the mobile robot, and calculates a current location coordinatesvalue of the mobile robot on the basis of a moving distancecorresponding to the movement velocity.
 12. A method of calculating alocation in a real-time manner using a combination of odometry andartificial landmarks, the method comprising: (a) detecting an imagecoordinates value of the artificial landmark corresponding to a locationin a two-dimensional image coordinate system with respect to a mobilerobot from an image obtained by photographing a specific space where theartificial landmarks are provided; (b) comparing a predicted image valueof the artificial landmark, obtained by converting a locationcoordinates value of the artificial landmark into an image coordinatesvalue corresponding to the location in the two-dimensional imagecoordinate system with respect to a location coordinates valuecorresponding to a location in an actual three-dimensional spatialcoordinate system of the mobile robot, with an image coordinates valuedetected by the landmark detection unit to detect the locationcoordinates value of the artificial landmark; (c) calculating a currentlocation coordinates value of the mobile robot using a predeterminedlocation calculation algorithm based on the image coordinates valuedetected in the (a) detection of the image coordinates value and thelocation coordinates value detected in the (b) comparison of thepredicted image value; (d) calculating a current location coordinatesvalue of the mobile robot using a predetermined location calculationalgorithm based on odometry information of the mobile robot; and (e)updating the current location coordinates value of the mobile robotusing the location coordinates value calculated in the (c) calculationof the current location coordinates value when the location coordinatesvalue calculated in the (c) calculation of the current locationcoordinates value exists, or using the location coordinate valueobtained in the (d) calculation of the current location coordinatesvalue when the location coordinates value calculated in the (c) does notexist.
 13. The method of claim 12, wherein the (e) updating the currentlocation coordinates value is repeated by performing processes ofcalculating the location coordinates value through the (c) calculationof the current location coordinates value based on the image coordinatesvalue or (d) calculation of the current location coordinates value basedon the odometry information after the current location coordinate valueof the mobile robot is updated, and updating the current locationcoordinate value of the mobile robot using the calculated locationcoordinates value.
 14. A computer-readable medium having embodiedthereon a computer program for performing the method of claim
 12. 15. Acomputer-readable medium having embodied thereon a computer program forperforming the method of claim 13.